Click any tag below to further narrow down your results
Links
Kestra 2.0 splits into a control plane and data plane, letting workers connect via gRPC instead of requiring direct database access, which means you can deploy them across regions and networks. The release also adds AI agent integration, a revamped no-code editor, and decouples the queue from the database for simpler configuration.
- Workers now connect to the control plane over a single outbound gRPC stream instead of needing direct database access, enabling deployment in different regions, clouds, or restricted networks without opening inbound ports.
- Queue and repository are now independent, so you pick them separately (Postgres, Kafka, Redis, or AMQP) instead of being locked into paired options; this simplified the codebase to one implementation instead of multiple backends.
- MCP Tool Trigger exposes any existing flow as a callable tool for AI agents like Claude, skipping the need for custom integration layers.
- AI Copilot is now a persistent sidebar with three modes (Edit, Plan, Ask) that refines flows iteratively rather than generating from scratch, and reads your namespace configuration to ground suggestions.
Since 2013, Buildkite has managed CI pipelines for firms like OpenAI, Airbnb, and Shopify and now orchestrates workflows for companies including NVIDIA, Meta, and Discord. It offers deep parallelization and fan-out orchestration for large-scale workloads, powering software used by over a billion people daily. A 30-day free trial is available.
- Buildkite has run CI for OpenAI, Airbnb, Canva, Uber, and Shopify for 7-12 years as they scaled.
- It now also orchestrates workflows for NVIDIA, Anthropic, Meta, xAI, Discord, Reddit, and others, plus tools like vLLM, AWS Lambda, and Bazel.
- Its core advantage is fan-out parallelization across agents rather than sequential or single-runner CI, speeding up large, complex builds.
- A 30-day free trial gives full access to this orchestration stack with no upfront cost.
Amazon MWAA runs Airflow pipelines on AWS as a managed service with serverless or provisioned deployment options. It handles infrastructure provisioning, scaling, security, and integrates with SageMaker for end-to-end ML and data workflows.
- MWAA now offers two deployment modes: serverless (pay-per-execution, auto-scaling) and provisioned (full config control with AWS-managed infrastructure)
- It integrates directly with the next-gen SageMaker Unified Studio, letting Airflow run alongside Jupyter notebooks for unified data prep, training, and inference pipelines
- Both modes run inside your VPC with end-to-end encryption and IAM controls, with serverless adding workflow-level permissions and provisioned offering environment-wide team separation
This article discusses a live session with Alejandro Aboy, a Senior Data and AI Engineer, about the evolving role of data engineers in the context of AI. Alejandro emphasizes that strong data engineering skills, particularly data modeling and orchestration, are crucial for successful AI work and shares practical steps for integrating AI into existing data workflows.
- Data modeling accounts for roughly 80% of the skill transfer from data engineering to AI engineering, according to Aboy.
- Good orchestration/workflow design can handle most AI tasks without needing complex agents.
- Precise column/model descriptions function as prompts for AI, so metadata and documentation quality directly determine AI performance.
- AI still lacks common-sense judgment (illustrated by a debugging agent missing context), so critically reviewing outputs rather than accepting them blindly remains essential.